Papers with automatic evaluation of machine translation
Metric for Automatic Machine Translation Evaluation based on Universal Sentence Representations (N18-4)
Copied to clipboard
| Challenge: | Sentence representations can capture information that cannot be captured by local features based on character or word Ngrams. |
| Approach: | They propose a supervised regression model using universal sentence representations capable of capturing information that cannot be captured by local features based on character or word Ngrams. |
| Outcome: | The proposed model achieves state-of-the-art performance with only sentence representation features . |
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization (2023.acl-long)
Copied to clipboard
| Challenge: | evaluators of machine translation systems often use text-based metrics to evaluate performance . however, these metrics lack semantic-level information and exhibit poor correlation with human ratings . authors propose a method to reduce inference bias of neural metrics in out-of-distribution data . |
| Approach: | They propose to reduce inference bias by using uncertainty estimation, test-time adaptation, and inference to reduce model uncertainty. |
| Outcome: | The proposed method reduces model uncertainty and improves correlation performance across models. |
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are limited and can be easily portable to new languages. |
| Approach: | They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. |
| Outcome: | The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models. |
LENS: A Learnable Evaluation Metric for Text Simplification (2023.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for text simplification are based on unitary or outdated models, making them unsuitable for this approach. |
| Approach: | They present a learnable evaluation metric for text simplification using language models . they also introduce a human evaluation framework that rates simplifications from several models a list-wise manner . |
| Outcome: | The proposed model correlates much better with human judgment than existing metrics. |